SYSTEM AND METHOD FOR DETECTING FACE 



BACKGROUND OF THE INVENTION 

This application claims the priority of Korean Patent Application No. 10-2002-0067974 
filed on November 4, 2002, in the Korean Intellectual Property Office, the disclosure of which is 
incorporated herein in its entirety by reference. 

1 . Field of Invention 

- The present invention relates to a system and method for detecting a face, and more 
particularly, to a system and method for detecting a face that is capable of quickly and correctly 
deciding, using an algorithm for determining whether a face is occluded, whether an input facial 
image is occluded. 

2. Description of the Related Art 

As the information society advances, automatic teller machines have rapidly come into 
wide use. Financial crimes in which money is illegally withdrawn using credit cards or 
passwords of other persons have also increased. To deter these financial crimes, a CCTV is 
installed in automatic teller machines to identify criminals. However, criminals often commit 
these crimes while wearing sunglasses or caps so as not to be photographed by the CCTV, and 
thus, it makes it difficult to identify the faces of the criminals. 

Korean Patent No. 0306355 (entitled "User identification system and automatic teller 
machine using the same") discloses a user identification system for identifying the face of a user, 
by acquiring a facial image of the user, obtaining a facial region through filtering of only a skin 
color region, extracting an eye position from the obtained region, setting a range of a mouth and 
nose on the basis of the eye position, and checking whether confirmable characteristic points 
exist. However, the eye position is extracted and the mouth and nose positions are then 
extracted on the basis of the extracted eye positions. Thus, if eye positions are not extracted, it 
is difficult to extract the mouth and nose positions. Further, image data of. the user is not 
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. searched but the presence of facial components (e.g., eyes, nose, mouth, and the like) is merely 
checked to identify the user. Thus, the face detection of a user cannot be correctly performed. 

In addition, Korean Patent No. 0293897 (entitled "Method for recognizing face of user of 
bank transaction system") discloses a method for recognizing the face of a user, which comprises 
5 the steps of determining facial candidate entities matching with an input user image using chain 
tracking, extracting contour points and comparing brightness values of the contour points to 
search graphics corresponding to eyes and mouth of the user, calculating a recognition index for 
the face, extracting only a single facial candidate entity, and comparing the recognition index for 
the face of the extracted facial candidate entity with a reference recognition index for the face. 

10 However, the presence of the eyes and mouth is determined according to whether the contour 
points of the eyes or mouth have been extracted. Thus, there is a possibility of misrecognizing 
sunglasses as eyes in a case where the sunglasses a user wears are similar to the eyes of the user 
in view of their shape and size. 

Furthermore, since conventional face detection technology uses color images, it is 

15 difficult to detect feature points for the nose and mouth of the user due to illumination. 
Consequently, the feature points of the nose and mouth of the user may not be detected. 
Therefore, there is also a problem in that the legal user maybe recognized as an illegal user. 

SUMMARY OF THE INVENTION 

20 " 

The present invention is conceived to solve the aforementioned problems. It is an object 
of the present invention to provide a system and method for detecting a face of a user that is 
capable of quickly and correctly deciding whether an input facial image is occluded although a 
variety of facial images under different conditions are inputted. 
25 According to an aspect of the present invention for achieving the object, there is provided 

a system and method for detecting a face of a user,, wherein it can be determined whether a facial 
image is occluded by extracting eigenvectors and weights from an input facial image of a user 
using PCA (Principal Component Analysis) and assigning the SVM (Support Vector Machines) 
to an algorithm for determining whether the image is occluded." 
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BRIEF DESCRIPTION OF THE DRAWINGS 



The above and other objects and features of the present invention will become apparent 
from the following description of an embodiment given in conjunction with the accompanying 
drawings, in which: 

FIG. 1 is a schematic block diagram of a system for detecting the face of a user 
according to the present invention; 

FIG. 2 is a flowchart illustrating the method for detecting the face of a user according to 
the present invention; 

FIG. 3 is a flowchart illustrating a method for authenticating a facial image to which the 
method for detecting the face according to the present invention is applied: 

FIG. 4 is a diagram illustrating an embodiment of the present invention in which the size 
of the facial image is normalized; and 

FIGS. 5a and 5b are pictures illustrating examples of a plurality of training images. 

DETAILED DESCRIPTION OF THE INVENTION 

An embodiment, of the present invention will now be described with reference to the 
accompanying drawings. 

FIG. 1 is a block diagram of a system for detecting the face of a user according to the 
present invention. The system for detecting the face of a user comprises a memory unit 100, a 
facial image recognition unit 200. and a facial image decision unit 300. 

The memory unit 100 stores eigenvectors and weights extracted from a plurality of training 
images, which are classified into a normal facial image class and an occluded facial image class. 
The normal facial image class is obtained by extracting eigenvectors and weights from normal 
facial images to which illumination changes, countenance changes, beards, scaling shift and 
rotation changes are applied and storing the extracted eigenvectors and weights, and the occluded 
facial image class is obtained by extracting eigenvectors and weights from partly occluded facial 
images to which illumination changes, countenance changes, scaling shift and rotation changes 
are applied and storing the extracted eigenvectors and weights. That is, the normal facial 



" images are identifiable facial images, and the occluded facial images are unidentifiable facial 

images because the faces are partly occluded with sunglasses, masks, mufflers and the like. 
Herein, the normal and occluded facial image classes are used to derive an occluding-decision 
algorithm , so as to decide whether the user, facial images have been occluded. The normal 
5 facial image class has a value of 1. whereas the occluded facial image has a value of -1. A 
process of deriving an algorithm for deciding whether facial images are occluded will be 
explained in detail with reference to the following mathematical expressions. 

The facial image recognition unit 200 functions to extract eigenvectors and weights from 
an input facial image using PCA to classify the normal and occluded face using Support Vector 
10 Machines and comprises a monochrome part 210. a facial image detection part 220, a facial 
image normalization part 230. a facial image division part 240, and an eigenvector/weight 
extraction part 250 using SVM. . 

The monochrome part 210 converts an input color image into a monochrome image. 
The reason for this is that since color and brightness components are mixed together in the color 
15 image configured in an RGB (Red. Green, Blue) mode, error due to brightness changes may be 
generated upon the extraction of the eigenvectors. 

The facial image detector 220 uses a technique for dividing the input image into 
background and facial regions, and detects the facial region from the input image using Gabor 
filters. Here, a method of detecting a facial region using the Gabor filters is performed by 
20 applying some sets of Gabor filters having various directionalities and frequencies into the input 
image and then detecting the facial region in accordance with response values thereof. 

The facial image normalization pan 230 performs corrections for image brightness due to 
illumination, facial image size due to distance from the camera, inclination of the facial image, 
and the like to normalize the facial region. 
25 The facial image division part 240 divides the normalized facial region into a higher 

region centered on the eyes and a lower region centered on the nose and mouth. Here, the 
reason for causing the facial region to be divided into the higher and lower regions is to quickly 
and correctly extract the eigenvectors of the respective facial components by restricting the size 
of the search region for each of the facial components since eigenvectors may be extracted from 
30 wrong region if the search region is too wide to extract the respective facial components. 
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Further, since peripheral regions are eliminated, noise components can be reduced. 

The eigenvector/weight extraction part 250 extracts eigenvectors and weights of the eyes, 
nose and mouth, which are major components of the face, using PCA according to the divided 
facial regions. Here, eigenvectors of the eyes, nose and mouth can be simultaneously extracted, 
because the facial region in which the eyes, nose and mouth are located is restricted upon 
extraction of the eigenvectors. 

Hereinafter, the mathematical expressions used for extracting the eigenvectors and 
weights using PCA will be described. 

Y = V\>h> r i^ r u-x> r u] (1) 
r=[r/ ? r 2 ' 3 r 3 ' v .. 5 r / ^/ 9 r / /] (2) 

[Formula 1 ] 

1 ,v 

;Vtf 

i M 

where F in Equation (1) represents facial images having a size of I x J stored in the 
normal facial image class of the memory unit 100, and F in Equation (2) represents facial 
images having a size of I x J stored in the occluded facial image class of the memory unit 100. 

Formula 1 is used to obtain an average facial image of the normal facial images and an 
average facial image of the occluded facial images, respectively. Here, N is the total number of 
normal facial images, and M is the total number of occluded facial images. 

First, a method of extracting eigenvectors and weights based on the normal facial images 
will be explained. 

T is applied to 4* in Formula 1 so. as to obtain the average facial image, and a vector 
O, is then calculated by subtracting the average facial image ( T ) from the facial images (T ). 
That is, <D, = T - T . 

Using the vector O, calculated as such, a covariance matrix is produced in accordance 
with Formula 2 below. 
[Formula 2] 
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Eigenvalues (A f . ) and eigenvectors ) can be calculated using Formula 2. In such a 

case, the eigenvalues are first calculated using the equation, Cx = Ax, and the eigenvectors are 
5 then calculated. 

Thereafter, weights can be calculated using the eigenvectors calculated as such in 
accordance with the following Formula 3. 
[Formula 3] 

Wf =(r- v F)x W . 

10 Using Formula 3, the weights ( w f ) are calculated. 

Although only a method of extracting eigenvectors and weights from the normal facial 
images has been described above, a method of extracting eigenvectors and weights from the 
partly occluded facial images is performed in the same manner as the method of extracting 
eigenvectors and weights from the normal facial images. Further, eigenvectors and weights are 
1 5 extracted from the higher and lower regions in the facial region, respectively. 

The facial image decision unit 300 decides whether the input facial image is occluded, 
through the hiding-decision algorithm that has been, obtained from the training images stored in 
-the memory unit 100. The occluding-decision algorithm, Support Vector Machines, is 
expressed as the following Formula 4. - 
20 . [Formula 4] 

/=! ... 

If a value obtained from the hiding-decision algorithm in Formula 4 is 1 , the facial image 
is decided to be a normal one. On the other hand, if the value from the algorithm is -1, the 
facial image is determined to be a partly occluded one. This is because the algorithm for 
25 deciding whether the facial image is occluded has been configured after the normal and partly 
occluded facial images stored in the memory unit 100 are set to have class values of 1 and -1, 
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respectively, and then trained. 

In Formula 4 above, y i , A i and b are set by substituting a set of the class value (1, in 

the present invention) of the normal training image, the eigenvectors and the weights, which are 
stored in the memory unit 100, and another set of the class value (-1, in the present invention) of 
5 the partly occluded facial images, eigenvectors and weights, into Formula 4, respectively. 
These values may vary as the training images stored in the memory unit 100 are updated. 
The formula for calculating the polynomial kernel K is expressed as follows: 
) = (<*■*, >)' 

where K can be calculated by an inner product of x and Xj (i.e., x-x s = |jc|| jc f - j cos(^) ) 9 x is 

10 the weights, Xj is the eigenvectors, and d is a constant. 

Thus, the class value f(x) of the facial images, which will determine whether the 
facial images are occluded, can be obtained by applying y. , X i and b obtained using the 
eigenvectors and weights of the training images and then applying the eigenvectors and weights 
extracted from the facial images to the polynomial kernel K(x, x i ) . 

15 Hereinafter, a process of calculating y. s k k and b of the hiding-decision algorithm 

will be described by way of example. 

In a case where 5 frames of normal facial images and 8 frames of partly occluded facial 
images are stored in the memory unit 100, a method of deriving the factors (y j , X. ( and b ) of 

the algorithm in Formula 4 is expressed as follows: 
/ 

20 1 = sign(^ i y i Jl j K(x,x i )-\-b) : First image of the normal facial image 

/ 

1 = sign( y ^y j A i K(x.x i )-^ b) : Second image of the normal facial images 

/ 

1 = sign(^ j y i X i K{x,x i )^b) : Fifth image of the normal facial images 
/ 

- 1 = sign(^y i /i i K{x, x t ) + b) : First image of the partly occluded facial images 
/=i 
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- 1 = signi^y^K^x.x^ + b) : Second image of the partly occluded facial images 
/=] 

/ 

-1 = sign( f ^ t y i A i K{x,x i ) + b) : Eighth image of the partly occluded facial images 

As described above, the class value f{x) is set to 1 and -1 for the normal and partly 
occluded facial images, respectively, and eigenvectors and weights of the respective facial 
images are then applied to K(x 9 Xg) so as to calculate y i 9 A. and b that satisfy the formulas. 

Therefore, it can be correctly determined whether the facial images are occluded, by 
extracting eigenvectors and weights from the input facial images and assigning the extracted 
eigenvectors and weights into the hiding-decision algorithm. 

FIG. 2 is a flowchart illustrating a method for detecting a face according to the present 
invention. Referring to the figure, steps SI 50 to SI 58 of deriving an algorithm for determining 
whether a face is occluded will be first discussed below. 

A facial region is detected from an input training image (SI 50). Here, the input training 
image includes normal facial images to which illumination changes, countenance expression 
changes, beards, scaling shift and rotation changes are applied, and partly occluded facial images 
to which illumination changes, countenance changes, scaling shift and rotation changes are 
applied. 

The detected facial region is normalized (SI 52), and the normalized facial region is then 
divided into a higher region centered on the eyes and a lower region centered on the nose and 
mouth (SI 54). Eigenvectors and weights of the respective facial components are extracted 
using PC A according to the divided facial regions arid are stored in the memory unit 100 (SI 56). 
Since the detection of the eigenvectors and weights in the facial region has been illustrated in 
detail with reference to Formulas 1 to 3, the detailed explanation thereof will be omitted. 

Next, the factors of the algorithm for determining whether the face is occluded are set on 
the basis of the stored eigenvectors and weights (S I 58). 

Steps SI 50 to SI 58 are to derive the algorithm for determining whether the face is 
occluded. After the hiding-decision algorithm has been derived, steps SI 00 to SI 20 of deriving 
the-hidLng-decision algorithm are not performed any longer. 



Hereinafter, the process of determining whether an input user image is occluded will be 
discussed as follows. 

If a user image is inputted (SI 00) 5 the input color image is converted into a monochrome 
image (SI 02) and a facial region is then detected from the monochrome image using the Gabor 
filter response (SI 04). Thus, the use of a. monochrome facial image can reduce improper 
recognition problems involved with facial color due to makeup, skin color, etc. 

Then, the size, brightness, inclination and the like of the facial image are corrected, and 
the facial region is then normalized (SI 06). The normalized facial region is divided into a 
higher region centered on the eyes and a lower region centered on the nose and mouth (SI 08). 
Consequently, the sizes of search regions in the respective facial components (the eyes, nose and 
mouth) are limited (FIG. 4). 

Preferably, an entire facial image (320*240) is normalized into 60*60, and an eye region 
is set to (0,5) to (60,30), i.e. 60*25, and a nose and mouth region is set to (0,25) to (60,60), i.e. 
60*35, in the normalized facial image, as .shown in FIG. 4. If the eye region and the nose and 
mouth region are set according to the above size, a region corresponding to (0,25) to (60,30) 
overlaps by about 10%. This is because the size and position of the eyes, nose and mouth vary 
according to the input facial image and the images of the eyes, nose and mouth detected in the 
corresponding regions may be disconnected in cases where the eye region and the nose and 
mouth region are too small. 

, Furthermore, since the respective facial components are extracted from the restricted 
region, the size of the search region is limited. Thus, time needed for extracting the 
eigenvectors of the facial components can be reduced and noise components such as hair and 
background can also be reduced. 

Using PC A on the basis of the divided facial regions, the eigenvectors and weights of the 
eyes are extracted in the higher region and the eigenvectors and weights of the nose and mouth 
are similarly extracted in the lower region (S 1 1 0). 

Next, the extracted eigenvectors and weights of the user image are applied to the hiding- 
decision algorithm that has been derived from steps SI 50 to SI 58 (SI 12). That is, the detected 
eigenvectors and weights of the user image are applied to /£(*.*,) . and the class values f(x) 



are then calculated by using the values of y j . and 6 obtained through the eigenvectors and 
weights of the training image. Here, the class value f(x) may be obtained according to the 
higher and lower regions, respectively. 

Thereafter, it is determined whether the class values f(x) of the higher and lower 
5 regions are 1 or -1. in order to determine whether the facial image is occluded (SI 14 and SI 18). 
Thus, if the value obtained through the hiding-decision algorithm is 1, it is determined that the. 
facial image is normal (SI 16). On the other hand, if the value obtained through the algorithm is 
-1, it is determined that the facial image is partly occluded (SI 20). 

In the meantime, the determination of whether the facial image is occluded is performed 

10 at the same time in the higher and lower regions of the user image. In such a case, if it is 
determined that any one of the higher and lower regions is a occluded facial image, it should be 
determined that the user's facial image is occluded. At this time, since it is simultaneously 
determined whether the facial images in the higher and lower regions are occluded, it can be 
more quickly determined that the facial image is occluded. 

15 FIG. 3 is a flowchart illustrating a method for authenticating a facial image to which the 

method for detecting a face according to the present invention is applied. It is first determined 
whether an input facial image of the user is occluded (S200). Here, the determination whether 
the input facial image of the user is occluded can be made in the same manner as steps SI 00 to 
S120 shown in FIG. 2. 

20 Accordingly, if the facial image of the user is determined to be normal, user 

authentication is performed (S202). If it. is determined that the facial image of the user is 
occluded, a warning message is transmitted to the user (S206). Then, operations of detecting 
the user's facial image and deciding whether the facial image, is occluded are performed again 
(S200). At this time, when it is determined three or more times that the facial image of the user 

25 has been occluded (S208). user authentication is rejected (S210). 

FIGS. 5a and 5b are pictures illustrating examples of a plurality of training images. In 
particular, FIG. 5a shows a normal facial image class including images under different 
illumination conditions, images with countenance thereof different from one another, images 
with beards therein, and images with scaling shift or rotation change included therein. Herein, 
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the eigenvectors and weights of the facial images, in which features of the facial components (the 
eyes, nose, mouth, etc.) changeable according to illumination changes, countenance changes, 
beards and face inclination are fully considered, are stored in the normal facial image class. On 
the other hand, FIG. 5b shows a occluded facial image class including images of users wearing 
5 sunglasses, mufflers, etc. The eigenvectors and weights of the facial images, in which features 
of the facial components changeable according to sunglasses, mufflers, masks, illumination 
changes, countenance changes, scaling shift, rotation change, etc. are fully considered, are stored 
in the occluded facial image class. Therefore, since facial images under all conditions are 
included in the form of the training images in the class, it can be correctly determined whether 

iO the input facial images are occluded. 

Tables 1 and 2 below show examples in which performance of the algorithm for 
deciding whether a facial image is occluded according to the present invention is tested using the 
memory unit 100 in which 3200 normal facial images, 2900 facial images of users wearing 
sunglasses, 4500 facial images of users wearing masks and mufflers, and additional images 

15 without any facial images, etc. are stored. 

Table 1 represents test results performed by extracting 100 eigenvectors respectively 
from the normal facial image class and the occluded facial image class. 
Table 1 





Number of Support Vectors 


Number of Errors Occurring 


Higher Region 


250 


0 


Lower Region 


266 


1 



20 That is. the determination of whether a facial image is occluded has been made at least 

200 times based on the extracted eigenvectors. As a result, it can be understood that a low error 

generation rate is obtained. 

Table 2 below represents results of the same test for occluded facial images as that for 

the normal facial image when results obtained when the determination for the normal image of 
25 whether the image is occluded show a 98% search rate. Table 2 shows test results for 370 facial 

images of users wearing sunglasses (i.e., in the higher region) and 390 facial images of users 
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wearing mufflers (i.e. in the lower region). 



5 Table 2 





Search Rate 


Improper Recognition Rate 


Higher Region 


95.2% 


2.4% 


Lower Region 


98.8% 


0% 



As a result of the test, it can be understood that the determination whether a user's facial 
region is occluded is made at the search rate of 95% or above and that improper recognition due 
to the hiding of a user's facial features is extremely low. 



Since facial images under all conditions capable of being produced with the training 
10 images are included, the hiding-decision algorithm that can be employed for all facial images 
under various conditions can be derived. Thus, a highly successful search rate can be obtained 
when it is determined whether the facial image of the user is occluded. 

According to the present invention constructed as such, there is an advantage in that it 
can be correctly and quickly determined through the hiding-decision algorithm whether input 
15 facial images of the users are occluded, even though a variety of facial images are inputted. 

Further, there are advantages in that since the input facial image is divided into a higher 
region and a lower region to restrict the search regions of the respective facial components, 
processing time needed for extracting eigenvectors and weights of the respective facial 
components can be reduced, noise components such as hair and background in the images can 
20 also.be reduced, and eigenvectors and weights of the eyes, nose and mouth in the respective 
relevant regions can be simultaneously extracted. 

Furthermore, there is another advantage in that any influence of the background or 
illumination can be reduced because of the use of monochrome images, and thus, improper 
recognition problems, which may occur upon extraction of the eigenvectors, can also be reduced. 
25 Although the present invention has been described in connection with the preferred - 

embodiment thereof shown in the accompanying drawings, they are mere examples of the present 
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invention. It can also be understood by those skilled in the art that various changes and 
modifications thereof can be made thereto without departing from the scope and spirit of the 
present invention defined by the claims. Therefore, the true scope of the present invention 
should be defined by the technical spirit of the appended claims. 



13 



